Repository navigation
Add stability_report: a one-call summary of a pipeline's stability - #61
Merged
Merged
Conversation
stability_report(metrics, compare="modeling") takes the output dict of a metrics Vset (or a dict_to_df DataFrame) and reports, per metric: - a summary of the results (count, mean, std, min, max) - the share of the variance explained by each perturbed Vset on its own, with the rest attributed to interactions - with compare=, how often each choice is best across the combinations of the other perturbations (win rate, mean rank) - the results by each Vset's choices - plain-language headlines The StabilityReport prints as markdown, renders in notebooks, saves as HTML or markdown, and keeps its tables as DataFrames. The README example shows it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Xy8Dg7fAqocHGzMqVCiSK
The header wordmark is now inline SVG with its ring of drops grouped, so on the home page the ring makes one eased turn with a gentle swell, rests and repeats, as on the talk's title slide (off for prefers-reduced-motion, and on the guide and API pages). The favicon is the o alone, its drops bolder and filling the box so it reads at tab size: favicon.svg, with a 64px PNG fallback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Xy8Dg7fAqocHGzMqVCiSK
Member
Author
|
Also in this PR: on the docs home page, the o of the header logo (the ring of drops) now makes one eased turn with a gentle swell, rests and repeats, like the talk's title slide (off for prefers-reduced-motion, and not on the other pages). The favicon is now the o alone, with bolder drops so it reads at tab size (favicon.svg plus a 64px PNG fallback). |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #61 +/- ##
==========================================
+ Coverage 93.46% 93.88% +0.42%
==========================================
Files 14 16 +2
Lines 1897 2176 +279
Branches 211 243 +32
==========================================
+ Hits 1773 2043 +270
- Misses 79 81 +2
- Partials 45 52 +7 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #60 (merge that first; this PR then retargets to master).
Summary
stability_report(metrics, compare="modeling")takes the output dict of a metrics Vset (or adict_to_dfDataFrame) and reports, separately for each metric:compare=, how often each choice (e.g. each model) is best across the combinations of the other perturbations (win rate, mean rank);modeling(76% of the variance)".The returned
StabilityReportprints as markdown, renders inline in notebooks, saves as HTML or markdown (report.save("stability.html")), and keeps its tables as DataFrames. Options:by(default: the last Vset, i.e. the metrics;Nonepools),bigger_is_better,wrt. The README example now ends with it.Testing
tests/test_report.py: variance shares on designs with known answers (pure main effect → 100%, pure interaction → 0% main effects), win counts andbigger_is_better,byhandling, an end-to-end pipeline, input checks, rendering and saving.🤖 Generated with Claude Code
https://claude.ai/code/session_013Xy8Dg7fAqocHGzMqVCiSK